1. Field of the Invention
This invention relates to a proposal for a user interface environment in terms of operations of issuing commands to a computer and, more particularly, it relates to a proposal for a user interface environment that allows the user to issue commands in an intuitive and easily understandable manner to a computer.
Still more particularly, the present invention relates to a proposal for a user interface environment that allows the user to operate various devices in an information space in an integrated, intuitive and easily understandable manner on user's computer. In such a user interface environment, the user can operates various devices in an information space in an intuitive and easily understandable manner on a computer display.
2. Related Background Art
As a result of technological innovations in recent years workstations and personal computers, which are compact general purpose computer systems with high added values and enhanced functional features have been developed and marketed at low prices. Such computer systems are almost ubiquitous in various research institutions including colleges and universities, commercial firms and offices and ordinary homes where people spend their daily lives.
One of the recent trends in the field of computer technology is down-sizing of computers. Particularly, compact and lightweight portable computers (mobile computers) have become very popular owing to the recent development of semiconductor technology that has made very small LSI (large scale integration) chips commercially available and boosted the processing capacity of computer processors.
Portable computers include so-called “notebook PCs” and palm-top sized computers, which are also referred to as “PDA (personal digital assistants)”. These and other computers can be operated with systems in an mobile environment so that the user can use his or her own computer almost anywhere, indoors, outdoors, in moving vehicles, etc.
Most computer systems provide the user with an interactive processing environment so that the computer system operates in response to the command issued to it by the user and the outcome of the processing operation is displayed on the display screen of the system. Currently, there is occurring a shift from the character-based user input environment or “the CUI (character user interface) environment” such as DOS (disk operating system) shell screen to the graphic-based user input environment or “the GUI (graphic user interface) environment”. In the GUI environment, a disk top unit simulating a computer system and numerous icons are provided for the display screen.
In the case of a disk top type computer provided with a GUI, all resource objects to be handled in the computer system including files are expressed by icons. The user can intuitively issue commands by directly operating icons on the display screen that represent respectively programs, data, holders and devices by means of a mouse and other I/O devices (with motions such as click, drag and drop, etc.). Therefore, the user is no longer required to memorize specific commands and learn how to handle them. Nor he or she is required to learn how to operate the keys of the keyboard for issuing commands, which is normally a very cumbersome task.
Another of the recent trends in the field of computer technology is interconnection. Computers may be interconnected typically by way of local interfaces such as serial ports and parallel ports or by way of network interface cards (NICs) so that they may be connected to each other by way of one or more than one telecommunication networks. From the viewpoint of interconnection of computers, telecommunication networks include LANs (local area networks) provided on single sites, WANs (wide area networks) realized by connecting LANs by way of dedicated lines and the internet which is a global telecommunication network realized by connecting various networks.
Interconnection of computers provides advantages including that information and computer resources are shared by a number of computer users, that information is delivered and disseminated highly efficiently and that a plurality of computers can be driven in a cooperative and concerted manner to carry out a common mission. For instance, an operation of processing a digital object can be carried out highly efficiently when a plurality of computers are made to cooperate in an intensive way.
Particularly, in recent years, the user can link his or her computer easily to a telecommunication network as a result of the advent of proximity telecommunication systems such as wireless LAN that is a wireless version of Ethernet and blue tooth[1]. The computers and peripheral devices (e.g., printers) installed in a building can mostly be connected to a telecommunication network so that they may be shared for operation by a plurality of users. Additionally, computer users can carry their own computers to any place they want and link them to a telecommunication network without difficulty.
In a distributed computing environment, the computers connected to a network are in a “transparent” state and they can be operated by means of an application program regardless if it is stored in a local storage disk or in a remote disk. In other words, the user is no long required to know the locations of resource objects such as programs and data because they can be exchanged seamlessly among computers.
The user can link his or her computer to a telecommunication network without any difficulty. However, if the user wants to specify a particular computer or peripheral device (target) to which data are to be transferred by way of the network, he or she has to know its ID (or its address) if it is located just in front of the user. In other words, even in a transparent and distributed computing environment, user operations are rather indirect and the related computers are not cooperative so that the user cannot act intuitively.
For example, if the user want to transfer an object between two or more than two computers in a currently available GUI environment, he or she is required to display a list of the devices linked to the network on an explorer screen and find out the icons simulating the respective computers before proceeding to a “drag and drop” operation.
A technique for solving the user operation problem may be the use of a real world-oriented user interface, which represents the technology for expanding the scope of the user operation from the “digital space” produced on a display screen of a computer to the space of the real world in an discontinued way. With such a real world-oriented interface, the display screen of the computer may be expanded to the surface of the disk on which the computer is placed and also to the surface of the walls of the room where the computer is located. In other words, the information space of the user expands beyond the limit of the digital space of the stand-alone computer to the entire room so that digital objects of the digital space of the computer and physical objects of the real world can be handled in an integrated manner.
For instance, “pick-and-drop[2]” is an expansion of user operation “drag- and drop” that is reliably being utilized in the GUI environment and represents the technology of advanced interactions for mapping the transfer of objects such as data among a plurality of computers into an intuitive motion in the real world.
With the “pick-and-drop” technique, seamless “pick-and-drop” operations can be realized as a result of cooperation of a plurality of disk top computers. For example, when an object is dragged by operating the user input device that may be a track ball or a stick of a disk top computer to move the pointer in the display screen beyond an edge thereof, the drag operation may be succeeded to the display screen of the adjacently located disk top computer so that the pointer of the latter display screen appears, dragging the object.
On the other hand, the pointer in the display screen of the former disk top computer disappears, while dragging the object. This operation of the pointer of the former disk top computer picking up an object is referred to as “pick” while the operation of causing the pointer of the latter disk top computer to appear in order to continuously drag the object is referred to as “drop”. Thus, the entity of the object (e.g., a data file) is transferred behind the screen between the two computers by way of the network. In other words, the succeeded drag operation accompanies a movement of the entity of the object. FIG. 1 schematically illustrate an “pick-and-drop” operation of an digital object of adjacently located computers.
As for “pick-and-drop” operations, refer to Japanese Patent Application Laid-Open Publication No. 11-53236 whose assignee is the applicant of the present patent application, specifically passages [0052] through [0059] and FIGS. 6 through 9 of the above patent document.
On the other hand, with “media blocks[3]”, the user can assign data such as images to blocks having respective IDs and move the data by carrying the corresponding blocks or rearrange the data by physically rearranging the latter. Therefore, blocks may be arranged in a dedicated slot or on a dedicated rail of a computer and data may be moved or rearranged by transferring corresponding blocks in the computer.
“Hyper-drag[4]” provides a user interface environment for the portable user computer to cooperate with a shared computer designed to use a desk top or a wall as display screen. For example, “hyper-drag” operations can be supported by introducing a camera-based object recognition system into the information environment. More specifically, the image of a computer display screen may be projected onto the surface of a disk top or a wall by means of a projector. Furthermore, both a projected image that is an object in the logical space of the computer and an object of the real world (that may be found on a disk top) may be recognized on the basis of a corresponding image picked-up by a camera and the motions of the objects may be traced so that the user can operate the computer interactively on the display screen that is now expanded to the disk top or the wall.
With “hyper-drag”, the digital space of a computer is expanded into the space of the real world. For example, a drag operation conducted on a user computer is made to continue on the plane where the computer is placed (e.g., the surface of a disk) (see FIG. 2) and the plane or the space containing it may be expanded to an wall of the workshop (see FIG. 3). In other words, the space of the workshop where the computer is placed can be used as if it were a the disk top input device of the computer to operate for the object.
As for “hyper-drag” operations, refer to Japanese Patent Application No. 11-316461, the assignee of which is the applicant of the present patent application
The above described known techniques make it possible to expand the digital space of a computer into the space of the real world so that operations for handling objects can be carried out easily and intuitively by making a plurality of computers cooperate.
However, the above known techniques are adapted to transfer data only once or only a block of data and necessary connections are established only through user operations. Nor any sustained connections are assumed for those techniques. In other words, the user is required to handle objects in a digital space or in the real space for each transfer of data and, once the object handling or the data transfer is over, the computer is disconnected from the related devices.
However, the user may retain interest in any of the devices after transmitting data and/or one or more than one commands to an external device. For example, he or she may want to monitor the operation of a device after issuing a command to it or add a command or operation for handling the device. Then, since the connections of the device are sequentially cut, the user is required to operates the computer almost incessantly in order to allow the it to continuously communicate with the device. In other words, it is not possible to keep the connection in a standby state until the device is brought into a predetermined certain state.
For example, while it is possible to specify an object of connection typically by using the technique of “pick-and-drop”, neither the computer nor the object is provided with means for showing the state of “being connected” or what it is connected to so that the object of connection should be referred to only by using its name except the moment when it is connected to the computer. In other words, it is necessary that the state of connection is visible to the user in order to establish a sustained connection by means of a physical operation like “pick-and-drop”.
With “pick-and-drop”, the stylus of a digitizer is used for picking up a displayed object from a display screen and drop it in another display screen. The technique of “pick-and-drop” resembles an operation of picking up a physical object and drop it. The stylus cannot be operated to handle an object that is out of the reach of the user. In other words, a remote information device cannot be picked up and dropped by remote control.
With the technique of “hyper-drag”, on the other hand, it is possible to expand an information space from the digital space of a computer to an operational space of the real world to realize a spatial continuity for the handling of information. As a result, a digital object and a physical object can be handled in a cooperative manner. More specifically, a digital object can be dragged and dropped onto a physical object to establish a new link between the physical object and the digital object and a digital object can be attached to a physical object and moved in the real space. Furthermore, a digital object can be drawn out of a physical object.
However, while an informational environment can be made to three-dimensionally continue in the real world with the technique of “hyper-drag”, the technique is not adapted to handle a physical object on a computer display screen digitally or by means of a command. For example, while a digital object may be attached or transferred to a physical object such as a video player, the recording or replay operation of the video player is not something that can be handled by a command issued by way of a computer.
In addition to the studies as cited above, a number of other studies have been reported in the field of real world-oriented interface. For example, there are known studies on objects and places that are provided with a physical (and hence visible) identifier so that information may be displayed or transferred by means of computer resources. Systems as proposed by Chameleon[5]. NaviCam[6], CyberGuide[7], UbiquitousLinks[8] and document[9] may be regarded as interfaces through which a block of data can be taken out from an object or a place.
Chameleon is a system for displaying information by means of a magnetic sensor depending on the position and the posture of a hand-held display and commands can be input by moving the display with the system.
NaviCam is a system adapted to use an ID that can be identified by image recognition to provide information according to the identified ID. Navicom can be combined with gyros and voice recognition.
CyberGuide is a system adapted to display guide information on a PDA according to the position of the user by means of an ID using ultraviolet rays. The data to be displayed is stored in the PDA in advance.
UbiquistousLinks is a system adapted to establish a link from an object or a place to a resource object on a WWW (world wide web) by using an ID tag or an ultraviolet ray ID. With such a system, data can be transferred and the linked destination can be shifted sequentially in the order of the identified Ids.
The document [9] proposes a system resembling UbiquitousLinks that can establish a link from an object to an electronic information source such as WWW by attaching a tag to the object.
Additionally, the systems proposed by “Infospuit” [10], “Extended Disk Top Environment [11] and document [12] are adapted to transfer or assign data to an object in addition to drawing out data from an object. They are also adapted to bidirectional data transfer.
Infospuit is a system where an object in the real world and a computer can exchange information by way of a device that can recognize two-dimensional codes such as a remote control unit. It is also provided with a feature of switching its operation depending on the destination of data transfer.
Extended Disk Top Environment is a system adapted to virtually arrange icons in space by recognizing the images picked up by a camera. With the system, any printer in the picked up image can be registered in advance and a printing operation can be carried out by dropping the icon.
Document [12] proposes a system adapted to recognize a place or an object by means of a two-dimensional matrix or an infrared ray ID and see a picture or a memo showing voiced information and paste it to the place or the object.
However, none of the above described real world-oriented interfaces assume sustained connections.
Meanwhile, the network environment for interconnecting computers have been remarkably improved in recent years and nowadays users are not required to pay particular efforts for connecting computers to a telecommunication network (previously described). Furthermore, many devices designed for use in ordinary daily lives are currently provided with information processing means to make themselves compatible with a telecommunication network. For example, video recorders, television sets, refrigerators and electric kettles are information-oriented and can be remotely controlled by means of computer and a network.
However, when the user tries to specify a particular electric home appliance on the network, he or she is required to know its ID (or its address) and how to issue a command to it. In other words, the operation of issuing a command by way of a network is an indirect one and cannot be conducted in an intuitive way.
Electric home appliances are provided with excellent user interfaces such as a control panel/button and ordinary users who do not know how to operate computers can issue commands to them and use them almost without learning anything about electronic control systems. However, remote control systems involving networks are not as convenient as those of electric home appliances.